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Describing  Network  Analytical  Capabilities 


Develop  descriptions  that  support  fair  evaluation  of 
current  or  potential  capabilities  to  address  network 
defense  needs  and  operational  cycles 

•  “How  does  it  fit”  not  “Is  it  good” 

•  Input  to  acquisition,  not  decision  for  them 

•  Methodical  and  impartial,  not  objective 

Supportive  of  network  security,  but  applicable 
somewhat  beyond  just  network  security 

•  Harvest  analyst  expertise 

•  Consideration  of  carry-over  effects 
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Phase  1 : A  Language  Model 

Nouns  -  forms  of  data  handled  by  the  capability 

•  Inputs 

•  Processing 

•  Results 

Verbs  -  primitive  actions  supported  by  the  capability 

•  Data  handling 

•  Process 

•  Analytic 

•  Presentational 

Adverbs  -  characteristics  of  the  capability 

•  Process 

•  Product 

Prepositions  -  scope  or  limitations  of  the  capability 
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Assessing  Data 

What  is  the  primary  data  handled  by  the  capability? 

What  is  secondary  data  handled  by  the  capability? 

What  is  supportive  data  handled  by  the  capability? 

What  primitive  operations  are  associated  with  each? 

How  well  are  the  operations  implemented?  What  is 
missing? 
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Example:  Sourcefire  IDS 

Primary  input:  Packet  data 

•  Collect,  Abstract,  Parse,  Alert,  Store,  Query,  Export 

Secondary  input:  Network  map 

•  Select,  Group,  Aggregate 

Supportive  input:  Signatures 

•  Import,  Alert,  Store,  Export 
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Input/Processing/Output 


Input:  what  data  does  the  capability  consume? 

Sourcefire  consumes  network  packets 

Process:  what  data  is  used  for  control  or  direction  of 
the  capability? 

Sourcefire  uses  signatures  and  network  configuration 
information 

Output:  what  data  is  produced  by  the  capability? 

Sourcefire  produces  alerts,  and  selective  packet  capture 
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Network  Level  of  Abstraction 


Many  capabilities  are  focused  on  particular  range  of 
protocols  and  behaviors 

IP  layer:  packet-based  analysis,  does  not  get  into  local 
behavior  and  only  infers  application  behavior  (e.g.,  SiLK) 

Application  layer:  message-based  analysis,  does  not  deal 
with  transport  mechanics  (e.g.,  analysis  of  email 
patterns) 


Software  Engineering  Institute 


Carnegie  Mellon 


©2011  Carnegie  Mellon  University  8 


Assessing  Operations 

What  locus  of  operations  forms  the  “core” 
functionality  of  the  capability? 

What  are  secondary  operations? 

What  are  supportive  operations? 

How  well  are  those  operations  implemented? 

How  scoped  is  the  intended  application? 

Rating  scheme:  0-5,  plus  n/a,  not  eval,  absent 


Software  Engineering  Institute 


Carnegie  Mellon 


©2011  Carnegie  Mellon  University  9 


Summarizing  Operational 
Gaps/Maturity 


Functional  categories 

Gap  Severity  Maturity 

Balance  functional 
maturity  vs.  capability 
gaps 

All  tools  have  gaps 

Goal  is  to  see  how  peaks 
and  valleys  match 
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Process  Adverbs 


Sourcefire  IDS: 

Operational 

Qualitiative 

Tactical 

Concise 
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Product  Adverbs 


Sourcefire  IDS: 

•  Not  Data-diverse 

•  Immediate 

•  Responsive 

•  Interoperable 

•  Documented 

•  Supported 

•  Trained 

•  Robust 

•  No  Workflow 

•  No  AAA 


Software  Engineering  Institute 


Carnegie  Mellon 


©2011  Carnegie  Mellon  University  12 


Prepositions 

Under  Conditions  (e.g.,  edge  vs.  transit) 

At  Size  /  scale  (e.g.,  enclave  vs.  enterprise,  days  vs. 
months) 

Of  Scope  (e.g.,  CND  vs.  network  ops) 

Within  Coverage  (e.g.,  sparse  vs.  complete) 

In  time  (e.g.,  interactive  vs.  batch  vs.  continuous) 
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Phase  2:  Process  Descriptions 


What  form  of  reasoning  should  the  model  support? 

•  Fused-source  intelligence 

•  C2/OODA? 

•  Forensic? 

•  Bayesian  hypothesis  testing? 

•  Abductive  pattern  matching? 
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Network  Analysis  Approaches 


collection 


c - \ 

observe 

V _ ) 


(cert 


Software  Engineering  Institute 


Carnegie  Mellon 


©2011  Carnegie  Mellon  University  15 


Analysis  Decomposed 


Vulnerability 


Incident  Response 


'CERT 
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Next  Steps 

Expand  initial  visual  results  into  fair  comparisons 

•  Spider  diagrams 

•  Input/Process/Output  tables 

•  Network  level  tables 

•  Operational  maturity/gaps 

Define  requirements  for  evaluation  process  using  model 

•  Team? 

•  Approach? 

•  Process? 

•  Outcomes? 

•  Threats? 

Tie  capabilities  to  process  needs 

•  Threshold  approach  (score  needs  to  be  X) 

•  Conditional  approach  (capability  must  include  Y) 

•  Descriptive  approach  (need  to  support  operations  Z) 

Reasoning  Support 
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